Partition testing with usage models

The fundamental statistical strategy of improving sampling efficiency through partitioning the population is applied to software testing. Usage models make it possible to apply this strategy to improve the efficiency of testing. The testing budget is allocated to the blocks of the partition, and the software is executed on the sample of uses drawn from each block or sub-population of potential uses. Usage models support many strategies for automated partitioning and generating test cases from the partitioned population. Two strategies are shown here with the efficiency gains demonstrated. 1. Statistical testing Statistical testing is the comprehensive application of statistical science to the solution of software testing problems. The population of interest is all possible uses of the software under test, and this population must be represented in a form that facilitates posing and solving problems about the software and the testing process. The representation we use is the Markov chain usage model, which is described below. Further information about Markov chain usage models is presented in [4], which is reprinted in these conference proceedings. The statistical technique presented here is the use of partitioning to improve sampling efficiency. There is a “partition testing” literature [2,6] in software engineering which varies somewhat in terminology and conclusions from the statistical literature on partition sampling [1,3]. In this paper we will follow the statistical terminology and apply it to two goals of software testing, the estimation of software reliability and failure detection potential. All possible uses of the software under test can be represented by a usage model [5,7]. This model has the structure of a discrete parameter, finite state, time homogeneous Markov chain. States of use of the software are represented as states of the Markov chain and transitions from one state of use to another are represented by state transitions of the Markov chain. Each state transition has a fixed probability of occurrence. Usage models have two distinguished states, a single invocation state and a single termination state; an arc from the termination state back to the invocation state makes the chain recurrent. Significant analytical results for usage models are readily available [8]. 2. Partition testing Sampling theory is a mature area of statistics. To take advantage of stratified sampling methods it is necessary to define the sample in terms of a mathematical partition of the population. Sample allocations are then calculated for the blocks of the partition. Testing software by using random samples of test cases drawn from blocks of a partition defined on the population of all test cases is called “partition testing.” In this paper testing using a random sample drawn from the unpartitioned population will be called “simple random testing” to emphasize that both are based on random samples. Under proportional allocation of tests to blocks, partition testing will always perform at least as well as simple random testing, in terms of variances of estimators and failure detection probability [1,3]. Moreover, partition testing may outperform simple random testing, depending on the nature of the population and the degree of insight used in creating the partitioning strategy; which is to say that results of a given quality can be obtained with less testing or that a given amount of testing will produce higher quality results. Partitions allocate the set of all possible uses of the software under test into blocks, each of which can be tested independently of the other blocks. Simple random testing can be viewed as a special case of partition testing consisting of a single block. The software engineering literature notes frustrations with partition testing that arise from difficulties in defining a mathematically correct partition, deriving analytical properties of partitions and test cases, expenses associated with constructing partitions and test cases for the partitions, and ineffective allocation of tests to blocks. As will be demonstrated below, these difficulties are resolved through application of graph theory and Markovian analysis to the usage model, thus facilitating greater efficiency in software testing through sampling theory. 2.1. Basic results The following are basic results of sampling theory [1]: • Optimal allocation of samples to blocks of the partition based on a fixed total sampling budget. • Optimal allocation of samples to blocks to achieve a given sample estimator variance. • Estimate of the percentage of the sample population that falls into a certain category. • Variance of the estimate of the percentage of the sample population that falls into a certain category. To apply the basic results the following items of information are needed: • A usage model representing all possible uses of the software, at some level of abstraction. • A partitioning rule in terms of that creates blocks. • Probabilities , where is the probability mass associated with block . • Estimated failure rates , where is the estimated failure rate of block . • The number of tests to be run , or the budget (allowable cost) . • If tests are to be allocated based on a fixed testing budget, then testing costs for each block, , are needed. 2.2. Optimal test allocation The optimal test allocation can be computed to minimize the variance of the software reliability estimator given a constraint on the total number of tests to be run or the total cost to be allowed in testing. The optimal allocation of tests to the blocks given a total fixed testing cost makes the assumption that , where is the cost of general testing overhead, is the cost of running a test from block , and is the number of tests run from block . Note that the optimal allocation of tests to blocks based on a fixed testing cost takes into account the fact that it may be more expensive to run tests from one block than another. The optimal test allocation for block is calculated by: This equation depends on the value of , which is not directly defined when is given. may be calculated as: The optimal allocation of tests for block given a fixed total number of tests is equivalent to testing where tests from all blocks have a fixed cost of one. The optimal allocation is computed by: Given a target variance , the minimum cost allocation of tests to achieve is computed in the same manner as the optimum allocation of tests given a fixed testing budget. The only difference is the equation used to compute : 2.3. Estimate of overall failure rate After the number of tests has been allocated to each block, test cases are selected randomly (generated from the usage model) based on the use distribution of each block. The tests are then run and , the number of failures observed during testing in block , is recorded. The overall failure rate of the software is estimated as , where is the estimated failure